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ABSTRACT 


For  estimation  of  a probability  density  function  f by  an  a^>lrlcal 
probability  density  function  fQ  based  on  a saaple  of  alas  n froa  f,  a 
useful  aeasure  of  distance  is  the  I^-norn  llfn  - £ 1 1 - <J[fn(x)  - f(x)]2dx)* 
Considerable  study  of  the  rate  of  naan  square  convergence  of  | | f#  - f | j 
to  0 has  taken  place.  This  paper  investigates  the  rate  of  almost  sura 
convergence  of  | |fft  - f| | to  0,  and  characterises  nonenta  E{ 1 1 fQ  - f 1 1*1) 
of  order  k > 2 aa  veil.  Application  to  certain  eatiaatloo  p rob lean  in 
nonprarnatrlc  Inference  la  discussed. 


MTIS  GXA4I 

dd:  tab 

Unannounced 

Justification 


Fhrasest  Convergence  in  Lj-nore;  Honparanatric  density  eatlaatlon 


1*  Introduction.  Consider  estimation  of  a probability  density  function 
f»  defined  on  the  reel  line,  by  an  enplrical  probability  density  function 
fQ  based  on  a random  saaple  of  also  n from  f . A natural,  useful  and 
interesting  measure  of  the  distance  between  fQ  and  f is  given  by 
ll*B  - *11  ■ (J[fn(x)  - f(x)]*dx)\  ||*||  denoting  the  nora  of  the  apace 
Lj  (■“**»  •)*  Indeed,  considerable  attention  in  the  literature  (see  review 
papers  by  Hegnan  (1972a, b)  and  Fryer  (1977))  has  been  devoted  to  the 
rate  of  m<m  equal*  convergence  of  | | f - f | | to  0,  that  la,  the  rate  of 

\ U 

convergence  of  E{||fn  - f| |2)  to  0.  Further,  the  almtet  sure  (a.s.) 
convergence  of  | |fQ  - f | | to  0 has  been  studied,  by  Nadaraya  (1973),  but 
without  investigating  the  rate.  This  latter  question  is  explored  in  the 
present  paper.  Also,  higher  momenta  E(||fn  - f| |2r)  are  characterised. 

Besides  having  intrinsic  interest,  the  results  have  application 
(see  K.  F.  Cheng  and  Serfling  (1979))  in  connection  with  estimation  of 
efficacy-related  parameters  such  as  T(f)  ■ Jf*(x)dx  arising  in  certain 


nonparametrlc  Inference  problems.  For  this  and  other  such  T(f),  the 


In  order,  then,  to  obtain  useful  characterisations 


(») 


p(n*||fn  - f||2  > «a)  - 0(«n). 


*n  4 0.  Finally,  to  characterise  the  a. a.  order  of  aagnltuda  of 
the  flue tati  one  of  I(fn)  - T(f) , we  require 

<c>  «M2  -a.a.  °<«<“»» 

where  g(n)  la  a function,  such  as  (lot  n)^  or  (log  log  n)\  tending  to  •. 

Ve  shall  obtain  properties  (A) , (B) , (C)  as  consequences  of  bounds 
on  the  amenta  B{  1 1 fQ  - f||2r}.  Thaorens  1 and  3 provide  suitable  boimda 
under  different  types  of  assunptlons  on  f and  f , and  Theorem  2 and  4 

II 

provide  corresponding  implications  germne  to  (A) , (B)  and  (C) . He  also 
obtain  a further  result  apropos  to  (C)  (and  (A))  by  a different  approach, 
utilising  the  connection  between  the  I^-norn  and  the  sup-nora  (Thsorea  5) 
He  confine  attention  to  estlaators  f of  the  ktmal  or  w indcu  type: 

H 

f (*)  - (nc  r1  ! K((x  - 0/0,  - < x < -, 

1-1  x n 

where  K(u)  la  a "kernel"  (somtlms  a probabUlty  density)  and  (c  } la 

n 

a "bandwidth"  sequence  of  positive  constants  tending  to  0. 

The  results  of  this  paper  adnlt  In  som  Instances  extensions  to 
higher  dlaanslonal  data  and  analogues  for  other  types  of  nonpar aaetrlc 
density  estimator  f . 

U 

2.  Properties  of  | I fQ  “ f | | • He  shall  develop  order  bounds  for  the 

aoaeats  B{ | | fQ  - f | |2r)»  for  r a positive  Integer,  by  the  following 
approech.  Using  the  elementary  Inequality  |a  ♦ b|k  i I^dal*1  ♦ |b|*) 
for  k a positive  Integer,  along  with  I If  - f 1 1 t ||f  - If _ 1 1 + | |Kf 


The  rlght-aost  term  in  (*)  in  slaply  the  r-th  power  of  thn  integrated 
nqunrnd  bias,  /CEfQ(x)  - f(x)]2dx,  for  which  the  behavior  is  known  froa 
studies  of  tbs  assn  square  error  of  1 1 fQ  - f 1 1 . See  Lsaaas  2 and  3 below. 
The  other  tera  on  the  right-hand  side  of  (*)  is  an  r-th  order  analogue 
of  the  integrated  variance.  The  following  reeult  adequately  characterises 
the  behavior  of  this  tera  for  the  present  purposes. 


LSMiA  1.  Let  r be  a positive  integer.  As< 
and  J|K(x) |dx  are  finite.  Then 


- 4 - 


By  Independence  of  the  Y 4(x)'s,  1 £ i i n,  the  expectation  In  the  lute- 

ufX 

grand  la  0 except  in  the  case  that  each  index  in  the  list  1^,  1^, 

appears  at  least  twice.  In  this  case  the  Quaker  of  distinct  ilmnti 


in  the  set  (i^»  , ...»  ir,  Jr>  is  s r.  It  follows  that  the  nuaber 

of  ways  to  choose  i^,  1^,  Jr  such  that  the  expectation  la  (2) 

is  ncnsero  is  0(nr).  Moreover,  these  noosero  expectetlons  are  uniformly 


0(c  r) . Hence 

U 


B{||f  - Bf  | |2r)  “ 0(n~2rnrc“r)  - 0((nc  )"r).  □ 


The  case  r - 1 has  been  derived  by  Nadar eye  (1974)  with  explicit 
constant  in  the  0(<)  expression.  Turning  now  to  | |EfQ  - f|J,  we  first 
cite  a further  result  of  Nadaraya  (1974),  namely  that 

(3)  ||Bfn-  f|  |2  -0(c2s),  n 

for  f belonging  to  W^,  K belonging  to  , and  s an  even  integer  i 2. 
Here  W#  denotes  the  set  of  functions  f(x)  having  derivations  of  s-th 
order  inclusively,  s i 2,  f^(x)  being  a bounded  continuous  LjC--,  •) 
function,  and  Hg,  a an  even  integer,  denotes  the  class  of  kernels  K(x) 
satisfying 

(*•)  K(*)  ■ B(-x) , /K(x)dx  • 1,  supx|K(x)|  < •, 

(4b)  /x*K(x)dx  -0,  HiSs-1, 

<4c)  /x*K(x)dx  * 0,  /x*|K(x)|dx  < •. 

The  possibility  of  (3)  under  milder  restrictions  on  the  smoothness 


of  f has  been  investigated  by  B.  P.  Cheng  and  Serfllng  (1979).  Let 
denote  the  set  of  functions  f either  having  bounded  continuous  dsrivative 


, • 

-v  - ••  V' 
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*,  “) , or  being  Lipschltz  on  (-«*,  •)  and  vanishing  off  a finite 
Interval.  Let  denote  the  dasa  of  kernels  K(x)  satisfying 


(5) 


K nonnegative, /K(x)dx  ■ 1,  /x2 K(x)dx  < «. 


, 


Vbr  f e Wj  and  K f (3)  holds  with  a • 1.  Further,  it  ia  shown  that 

||Mn  - fj|2  ■ 0(cn^2q+*^q+1^)if  f is  Lip(-»,  •)  and  satisfies  the  q-th 
order  tail  restriction  /|x|>tf (*)dx  ■ 0(t”q),  t » (implied  by 
/|x|qf(x)dx  < •),  and  if  K belongs  to  H^.  In  other  words,  (3)  holds 
with  a given  by  (q  + *i)/(q  + 1).  Therefore,  for  s e (*i,  1),  we  define  W# 
to  consist  of  functions  f which  are  Lip  (-*»,  «)  and  satisfy  the  q-th 

order  tail  restriction  for  q - (s  - »0/(l  - s).  Also,  for  s < (%,1), 

we  define  H to  be  identical  with  H. . 

8 X 

Combining  all  these  results  on  | | Ef^  - f||,  we  have 

LEMMA  2.  Let  e e (S,  1)  or  be  an  even  integer.  Let  f e H(  and 

K c H^.  Then  (3)  holds. 

Let  us  now  utilise  Lemmas  1 and  2 In  conjunction  with  (*).  For 
cQ  - An  ' ' the  bounds  provided  by  these  lemnas  for  the  terms  in 

(*)  have  the  same  order,  0(n  2*r/(2s+l)j^  Therefore,  we  have 

THEOREM  1.  Let  s e (*j,  l)  or  be  an  even  integer , end  aeeme  f « W# 
and  K e H#.  Let  r be  a poeitive  integer.  Then 

<*>  B{ 1 1 fn  - f | |2r)  - 0((ncn)‘r)  + 0(cn2") 

and,  in  particular,  for  c ■ ^“I/Wa+I)* 

n 

E(||fn  - f||2r)  - 0(n-2*r/<2rfl). 





. • » i 


r 
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For  the  case  • an  even  Integer  and  r - 1,  relation  (6)  haa  been  given  by 
Nadaraya  (1974)  with  explicit  conatanta  In  the  0(*)  expreaalona.  Thoae 
conatanta  were  crucial  to  hla  purpose  of  eatlaatlng  the  optical  conatant 
A to  be  ueed  In  the  choice  cq  ■ An  ^(2e+l)^  por  our  purpose*,  however, 
theae  conatanta  are  Irrelevant  but  It  la  important  that  r nay  be  choeen 
arbitrarily  large.  By  virtue  of  thia  feature,  we  are  able  to  eatabliah 

THEOREM  2.  let  e c (*a , 1)  or  bo  an  ovon  intogor.  Assume  f e W , 

K c H#,  and  cn  ■ An'1^2*^.  Thom 

(I)  For  a < a/(2a  + l), 

<7, 

(II)  For  aQ  givon  by  Bn”®,  with  B oonetant  and  0 < (a  - %)/(2a  + 1), 

(8)  F(n%||£ft-  f| |2  > an)  - 0(an). 

PROOF.  Let  e > 0 be  given.  Let  r be  a positive  Integer.  Applying 
Theorem  1,  we  have 

<»  P(na||fn  - (||  > c)  s c'In2“rE{||fIl  - f||2rl 

For  o < a/ (2a  + 1) , (7)  thua  follow*  by  the  Borel-Cantelll  leeu  end  the 
fact  that  r may  be  chosen  arbitrarily  large.  Mow  replace  e by  s^  and  a by 
% In  (9)  and  obtain 

(10)  F(nS||fn  - f||2  > an)  s e“rO(nrC%“2,/(2,+1):i). 

For  aQ  * Bn  ®,  the  right-hand  aide  of  (10)  la  aean  to  be  0(aQ)  under  the 
condition  that 


' 
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e < 


JL.  !=iL. 
r+1  2s+l* 


Por  0 < (s  - h)/(2s  + 1) , this  condition  is  satisfied  for  a sufficiently 
large  choice  of  r and  thus  (8)  follows.  □ 

REMARK.  Regarding  the  properties  discussed  in  Section  1,  we  see 

froai  Theorem  2(1)  that  properties  (A)  and  (C)  hold  for  f c V(  and  K < H|( 

where  a satisfies  s/ (2s+l)  > b,  or  equivalently  s > S.  That  is,  (A)  and 

(C)  hold  for  all  possibilities  of  f and  K covered  by  the  conditions  of 

the  theorem.  Further,  property  (B)  holds  for  a of  the  form  Bn-0  with 

n 

8 K (■  " b)/(2s  + 1) ; as  a ■*  •»,  8 may  be  Increased  toward  h,  which  corres- 
ponds to  the  optimal  Berry-Essden  rate  in  typical  clrcuMtances.  □ 


: 


1 


I 

| 


* 


Up  to  now  we  have  considered  the  behavior  of  | |fn  - f | | under  a 
range  of  smoothness  and/or  tall  restrictions  directly  Imposed  on  f (and 
with  K satisfying  compatible  restrictions) . Let  us  now  address  the  question 
r*l*tlve  to  restrictions  on  the  characteristic  function  of  f , following 
Parsen  (1962)  and  Watson  and  Leadbetter  (1963) . The  characteristic  function 
if  1*  a*id  to  decrease  algebraioally  of  degree  p > 0 If 

0 * U*|t|-H.lt|P|*f(t)l  < *• 

The  compatible  estimator  fQ  la  of  kernal  type  and,  moreover,  must  be  of 
"algebriao  form" : K must  be  square  Integrable  and  Its  Fourier  transform 
•K  aust  be  a bounded  even  square  Integrable  function.  As  a parallel  to 
Lease  2,  we  have 

LEMfA  3.  Let  decrease  algebraically  of  degree  p > 0,  and  let 
be  of  algebraic  form  with  K euah  that  J|t|”2p[l  - tL,(t)]2dt  < • <m 
with  c ■ (^-l/2p. 

u 


s 


(11)  ||Efn  - f||2  - 0(n~(2p"1)/2p). 

In  fact,  Watson  and  Leadbetter  establish  (lx)  for  E{||fQ  - f | | 2 ) ana 
with  an  explicit  constant  in  the  0(*)  expression.  Their  result  yields 
(11)  by  Jensen's  inequality  and  Pub In 1 's  theorem. 

Combining  Lemmas  1 and  3,  we  easily  obtain 

THEOREM  3.  Let  decrease  algebraically  of  degree  p > 0 and  let 
fQ  be  of  algebraic  form  with  K bounded , absolutely  integrable , and  satis- 
fying /{ t | 2p[l  - ♦ (t)]2dt  < • and  with  c - Cn_1^2p.  Let  r be  a poei- 

lv  XI 

tive  integer.  Then 

* E{||fn  - f||2r}  - 0(n=r(2p"1)/2p). 

The  preceding  result  is  an  analogue  to  Theorem  1.  A corresponding 
analogue  to  Theorem  2,  by  a similar  proof  as  before,  is  the  following. 
THEOREM  4.  Assume  the  conditions  of  Theorem  3.  Then: 

(i)  For  a < (2p  - l)/4p,  (7)  holds. 

(ii)  For  aQ  given  by  Bn"8,  with  B < (p  - l)/2p,  (8)  holds. 

Regarding  properties  (A),  (B)  and  (C)  of  Section  1,  comments  similar 
to  the  Remark  following  Theorem  2 apply.  That  is,  (A)  and  (C)  hold  if 
p > as  p -*•  •,  (B)  holds  for  8 arbitrarily  close  to  S,  the  optimal  value. 

We  now  augment  the  assertions  of  Theorems  2(1)  and  4(1)  with  an 
additional  result  utilising  the  following  lemma  from  Serf ling  (1979). 

Here  we  denote  by  11*11,,  the  aup-norm,  | |h| |„  - sup^  |h(x)|,  and  by 
1 1 • I lx  th*  I |t>l  I x * J|h(x)|dx. 


I 
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LEMMA  4.  Let  t be  a probability  density  on  (-*»,  -)  satisfying 
J |x|>t.f(x)dx  ■ 0(t”**) , -*■  • implied  by  /|x|^f(x)dx  < •) . Let  gn  be  a 
sequence  of  probability  density  functions  satisfying  | jgn  - f | |w  -*•  0. 

Th0n  Il9n  - f||j  • 0(ll9n  “ f| |^9/(q+1>) # further,  f vanishes  off 
a finite  interval , then  ||gn  - f||x  - o ( 1 1 f n - f||j. 

Various  theorems  in  the  literature  provide  conditions  on  f and  on  the 
•■timator  (in  some  cases  considering  fR  other  than  the  kernel  type) , 
under  which  the  rate  of  a.s.  convergence  of  I If  - f 1 1 to  0 is  charac- 
terixed i for  soma  y > 0, 

<«>  »Tllf„  - »• 

In  particular,  under  minimal  smoothness  conditions  on  f and  for 
suitable  K,  results  of  Reiss  (1975),  Silverman  (1978)  snd  Winter  (1978) 
establish  that  (12)  holds  for  y * 1/3.  Using  such  results  in  conjunction 
with  Lemma  4 and  the  inequality  | |fQ  - f| | s I l£n  ~ fl lx  * I lfn  “ fl l.» 
we  obtain  corresponding  results  for  the  L2-norm.  We  thus  have 

THEOREM  5.  Let  t and  fQ  satisfy  conditions  under  which  (12)  holds 
for  same  y > 0.  Let  f also  satisfy  the  q-th  order  tail  restriction  of 
Lmtna  4,  where  q ■ • denotes  that  f has  bounded  support.  Then,  for 
a < y(2q  + l)/2(q  + 1),(7)  holds. 

Let  us  briefly  compare  the  rates  provided  by  Theorems  2(1),  4(1)  and 
5.  In  Theorea  5 let  us  take  y approximately  l/3(y  < 1/3)  and  q approxi- 
mately 1 (q  > 1).  Then  (7)  with  a approximately  h follows.  In  this  case 


i 


i 


the  assumptions  of  Theorem  2 are  comparable  for  s ■ 3/4,  yielding  (7) 
with  a approximately  3/10.  Likewise  the  assumptions  of  Theorem  4 are 
comparable  for  p ■ 2,  yielding  (7)  with  a approximately  3/8.  Thus  Theorea 


4 !•  non  effective  in  this  comparison.  Furthermore,  Theorems 
2 end  4 (and  their  foundations  Theorems  1 and  3)  provide  additional  Infor- 
mation on  ||fn  - f||.  Of  course,  the  approach  based  on  Lemma  4 can  also 
be  exploited  to  provl.de  bounds  on  E{  1 1 fQ  - f||2r},  via 

E<l|fn  - *l|2r>  * D£>q  «||*B  " f|ll(2q+1)/<<1+1)}. 

for  a constant  D,  depending  only  on  f and  q.  This  requires  knowledge 

1 tH 

of  the  momenta  of  ||fQ  - f| | , a topic  which  has  received  little  attention 
In  the  literature.  Leadbetter  (1963)  has  established  B{||fQ  - f | |2}  • 

0(n  1+1^p_e)  under  conditions  similar  to  those  of  Theorems  3 and  4. 
However,  the  higher  moments  of  ||fn  - f | | ^ have  not  been  Investigated. 
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AB^RACT 

For  estlaatlon  of  a probability  density  function  f by  an  empirical  probability 
density  function  f^  baaed  on  a sample  of  slse  n from  f,  a useful  measure  of  distance 
la  the  L2-norm<44f^  — f|  l - f(a)]2dx)\  Considerable  study  of  the  rate 

of  mean  square  convergence  of  fp^fj^  toO  has  taken  place.  This  paper  Investigates 
the  rate  of  almost  sure  convergence  of  (jjf^  - Tf]  to  0*  and  characterises  moments 
E*M£n  " f I lh^  ot  order  k > 2 as  well.  Application  to  certain  estimation  problems 
In  nonpar ame trie  Inference  la  discussed. 
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